Portable device and operation method for tracking user&#39;s viewpoint and adjusting viewport

ABSTRACT

A portable device includes a display module, an image capture module, and an eye-tracking module. The image capture module is used to capture images steadily. The eye-tracking module is used to track a user&#39;s viewpoint position relative to the portable device. Viewport within the images captured by the image capture module is adjusted in accordance with the user&#39;s viewpoint position for generating modified images displayed on the display module instantaneously. The modified images shown on the display module can fit the background scene seen by the user, and the augmented reality experience on the portable device may be improved accordingly.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a division of application Ser. No. 15/368,693 filedon Dec. 5, 2016, which is incorporated by reference herein in itsentirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a portable device and an operationmethod thereof, and more particularly, to a portable device including animage capture module and a display module and an operation methodthereof.

2. Description of the Prior Art

In recent years, portable devices, such as smart phones, tablets,handheld game consoles, and car navigation systems, become popularbecause of the lightweight and compact displays and other additionalfeatures, such as wireless internet connection and image capture. Theportable device equipped with a display panel and a camera may be usedto perform an augmented reality (AR) function. Augmented reality is alive direct view of a physical and real-world environment whose elementsare augmented or supplemented by computer-generated sensory input suchas sound, video, graphics or other information. Augmentation isgenerally in real-time with environmental elements. With the ARtechnology, the information about the surrounding real world of the userbecomes interactive and digitized. However, the AR experience of theuser may be compromised because the image shown on the display isgenerally constrained by the specification of the camera and cannotalways fit the background scene when the viewing distance between theuser and the display becomes different.

SUMMARY OF THE INVENTION

A portable device and an operation method thereof are provided in thepresent invention. An eye-tracking module in the portable device is usedto track a user's viewpoint position relative to the portable device foradjusting viewport within images captured by an image capture module inaccordance with the user's viewpoint position for generating modifiedimages shown on a display module. Accordingly, the modified images shownon the display module may fit the background scene seen by the user, andthe augmented reality experience on the portable device may be improved.

According to an embodiment of the present invention, an operation methodof a portable device is provided. The operation method includes thefollowing steps. Images are captured steadily by an image capture moduleof a portable device. A user's viewpoint position relative to theportable device is tracked. Viewport within the images captured by theimage capture module is adjusted in accordance with the user's viewpointposition for generating modified images. The modified images aredisplayed on a display module of the portable device instantaneously.

According to an embodiment of the present invention, a portable deviceis provided. The portable device includes a display module, an imagecapture module, and an eye-tracking module. The image capture module isused to capture images steadily. The eye-tracking module is used totrack a user's viewpoint position relative to the portable device.Viewport within the images captured by the image capture module isadjusted in accordance with the user's viewpoint position for generatingmodified images displayed on the display module instantaneously.

These and other objectives of the present invention will no doubt becomeobvious to those of ordinary skill in the art after reading thefollowing detailed description of the preferred embodiment that isillustrated in the various figures and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic drawing illustrating an operation method of aportable device according to an embodiment of the present invention.

FIG. 2 is a stereoscopic schematic drawing illustrating a portabledevice according to an embodiment of the present invention.

FIG. 3 is a cross-sectional diagram of the portable device according tothe embodiment of the present invention.

FIG. 4 is a schematic drawing illustrating a step of adjusting viewportwithin an image captured by an image capture module according to anembodiment of the present invention.

FIG. 5 is a schematic drawing illustrating the difference between animage shown on a display module of a conventional portable device and animage shown on a display module of the portable device according to theembodiment of the present invention.

FIG. 6 is a schematic drawing illustrating components in the portabledevice and interaction between the components according to theembodiment of the present invention.

DETAILED DESCRIPTION

In the following detailed description of the invention, reference ismade to the accompanying drawings, which form a part hereof, and inwhich is shown, by way of illustration, specific embodiments in whichthe invention may be practiced. These embodiments are described insufficient detail to enable those skilled in the art to practice theinvention. Other embodiments may be utilized and structural changes maybe made without departing from the scope of the present invention.

The following detailed description is, therefore, not to be taken in alimiting sense, and the scope of the present invention is defined onlyby the appended claims, along with the full scope of equivalents towhich such claims are entitled. One or more implementations of thepresent invention will now be described with reference to the attacheddrawings, wherein like reference numerals are used to refer to likeelements throughout, and wherein the illustrated structures are notnecessarily drawn to scale.

Please refer to FIG. 1, FIG. 2, and FIG. 3. FIG. 1 is a schematicdrawing illustrating an operation method of a portable device accordingto an embodiment of the present invention. FIG. 2 is a stereoscopicschematic drawing illustrating the portable device in this embodiment.FIG. 3 is a cross-sectional diagram of the portable device in thisembodiment. As shown in FIGS. 1-3, an operation method of a portabledevice 100 includes the following steps. The portable device 100including a display module 10, an image capture module 20, and aneye-tracking module 30 is provided. The portable device 100 may be asmart phone, a tablet, a handheld game console, or a car navigationsystem, but not limited thereto. In step S11, images 90 are capturedsteadily by the image capture module 20 of the portable device 100. Instep S12, a user's viewpoint position relative to the portable device100 is tracked by the eye-tracking module 30. The step S11 and the stepS12 may be performed simultaneously. In step S20, viewport within theimages captured by the image capture module 20 is adjusted in accordancewith the user's viewpoint position for generating modified images. Instep S30, the modified images are displayed on the display module 10 ofthe portable device 100 instantaneously. In other words, the imagecapture module 20 is used to capture the images steadily. Theeye-tracking module 30 is used to track the user's viewpoint positionrelative to the portable device 100. Preferably, the step of capturingthe images by the image capture module 20, the step of tracking theuser's viewpoint position, the step of adjusting the viewport within theimages captured by the image capture module 20 in accordance with theuser's viewpoint position for generating the modified images, and thestep of displaying the modified images on the display module 10 may beperformed simultaneously for presenting a real-time and live direct viewof the environment around the user.

As shown in FIG. 2, the viewpoint of the user is different from theviewpoint of the image capture module 20, and the images captured by theimage capture module 20 and displayed on the display module 10 will notfit the background scene seen by the user without modifying the capturedimages especially when the distance between the user and the portabledevice 100 changes. However, by the operation method in this embodiment,the modified images displayed on the display module 10 can fit thebackground scene behind the portable device 100 and seen by the user.

As shown in FIG. 1 and FIG. 3, in some embodiments, the step of trackingthe user's viewpoint position may include tracking a distance and anorientation between at least one eye E of the user and the portabledevice 100, and calculating a distance and an orientation between atleast one eye E of the user and the image capture module 20 of theportable device 100. The distance and the orientation between at leastone eye E of the user and the image capture module 20 of the portabledevice may be calculated in accordance with a distance and anorientation between the eye E of the user and the eye-tracking module 30and a relative position difference between the eye-tracking module 30and the image capture module 20. For example, ΔEyeCam stands for thedistance and the orientation difference between the eye E and the imagecapture module 20, ΔEyeSen stands for the distance and the orientationdifference between the eye E and the eye-tracking module 30, ΔSenCamstands for the distance and the orientation difference between theeye-tracking module 30 and the image capture module 20, and ΔEyeCam maybe calculated by an equation listed below.ΔEyeCam=ΔEyeSen+ΔSenCam=(X_(ΔEyeSen)+X_(ΔSenCam),Y_(ΔEyeSen)+Y_(ΔSenCam),Z_(ΔEyeSen)+Z_(ΔSenCam))

In the above equation, X_(ΔEyeSen) stands for a position differencebetween the eye E and the eye-tracking module 30 in X axis, andX_(ΔSenCam) stands for a position difference between the eye-trackingmodule 30 and the image capture module 20 in X axis; Y_(ΔEyeSen) standsfor a position difference between the eye E and the eye-tracking module30 in Y axis, and Y_(ΔSenCam) stands for a position difference betweenthe eye-tracking module 30 and the image capture module 20 in Y axis;Z_(ΔEyeSen) stands for a position difference between the eye E and theeye-tracking module 30 in Z axis, and Z_(ΔSenCam) stands for a positiondifference between the eye-tracking module 30 and the image capturemodule 20 in Z axis, wherein ΔSenCam is fixed in the portable device100, and ΔEyeSen may be measured by the eye-tracking module 30.

In some embodiments, the step of tracking the user's viewpoint positionmay include tracking a distance and an orientation between each eye ofthe user and the portable device 100. Additionally, the image capturemodule 20 may include 3D image capture module, and the modified imagesdisplayed on the display module 10 may include naked eye 3D images. The3D image capture module mentioned above may include a 3D depth cameraconfigured to record position information of objects in the imagescaptured by the image capture module 20. Additionally, the displaymodule 10 may be a naked eye 3D display module, but not limited thereto.Specifically, the images captured by the image capture module 20 mayinclude different images for the left eye and the right eye of the user,and the modified images may include different images displayed on thedisplay module 10 for the left eye and the right eye of the user.

Please refer to FIG. 1, FIG. 3, and FIG. 4. FIG. 4 is a schematicdrawing illustrating the step of adjusting viewport within an imagecaptured by the image capture module 20 according to an embodiment ofthe present invention. As shown in FIG. 1, FIG. 3, and FIG. 4, an image90 is captured by the image capture module 20, and a first viewport V1within the image 90 is formed in accordance with the user's viewpointposition for generating and/or rendering a modified image. When theportable device 100 becomes closer to the user's eyes, the viewportwithin the image is adjusted to be a second viewport V2 shown in FIG. 4.Accordingly, the viewport within the captured image becomes smaller whenthe distance between the user and the portable device 100 is reduced,and the magnification is increased when the distance between the userand the portable device 100 is reduced. In other words, the capturedimages are trimmed for generating and/or rendering the modified imagesdisplayed on the display module 10 in accordance with the informationabout the user's viewpoint position relative to the portable device 100which is tracked by the eye-tracking module 30. The modified imagesdisplayed on the display module 10 present a simulated viewpoint similarto that of the user instead of the viewpoint of the image capture module20.

Please refer to FIG. 3 and FIG. 5. FIG. 5 is a schematic drawingillustrating the difference between an image shown on a display module10′ of a conventional portable device 100′ and an image shown on thedisplay module 10 of the portable device 100 in this embodiment. Asshown in FIG. 3 and FIG. 5, the image displayed on the display module 10is the modified image generated by the operation method of the presentinvention and the image displayed on the display module 10 fit thebackground scene BS. However, the image displayed on the display module10′ of the conventional portable device 100′ does not fit the backgroundscene. Therefore, the user's augmented reality experience on theportable device 100 may be improved.

Please refer to FIG. 3 and FIG. 6. FIG. 6 is a schematic drawingillustrating components in the portable device 100 and interactionbetween the components according to an embodiment of the presentinvention. As shown in FIG. 3 and FIG. 6, the display module 10 and theeye-tracking module 30 may be disposed at a front side of the portabledevice 100, and the image capture module 20 may be disposed at a backside of the portable device 100. The eye-tracking module may include aninfrared sensor, or other suitable sensors. As shown in FIG. 3 and FIG.6, the portable device 100 may further include a viewpoint calculationmodule 31, a viewport adjust module 32, a 3D modeling module 21, and a3D rendering module 11. The viewport calculation module 31 is connectedto the eye-tracking module 30 and used to calculate the distance and theorientation between at least one eye E of the user and the image capturemodule 20 of the portable device 100. The 3D modeling module 21 isconnected to the image capture module 20 for processing the image datafrom the image capture module 20. The viewport adjust module 32 isconnected to the viewport calculation module 31 and the 3D renderingmodule 11. The viewport adjust module 32 is used to adjust the viewportwithin the images captured by the image capture module and input therelation information to the 3D rendering module 11 for generating themodified images.

To summarize the above descriptions, in the portable device and theoperation method thereof in the present invention, the user's viewpointposition relative to the portable device is tracked and calculation foradjusting the viewport within the images captured by the image capturemodule and generating the modified images displayed on the displaymodule. The modified images displayed on the display module may fit thebackground scene seen by the user, and the augmented reality experienceon the portable device may be improved accordingly.

Those skilled in the art will readily observe that numerousmodifications and alterations of the device and method may be made whileretaining the teachings of the invention. Accordingly, the abovedisclosure should be construed as limited only by the metes and boundsof the appended claims.

What is claimed is:
 1. An operation method of a portable device,comprising: capturing images steadily by an image capture module of aportable device; tracking a user's viewpoint position relative to theportable device; adjusting viewport within the images captured by theimage capture module in accordance with the user's viewpoint positionfor generating modified images; and displaying the modified images on adisplay module of the portable device instantaneously, wherein themodified images displayed on the display module fit a background scenebehind the portable device and seen directly by the user without theportable device, and the user sees the background scene behind theportable device not through the display module, wherein magnificationsof the images captured by the image capture module are adjusted in orderto be displayed on the display module and blend with the backgroundscene seen by the user at the same time, and the magnifications of theimages captured by the image capture module are increased when adistance between the user and the portable device is reduced, whereinthe step of tracking the user's viewpoint position comprises: tracking adistance and an orientation between each eye of the user and theportable device, wherein the image capture module comprises a 3D imagecapture module, and the modified images displayed on the display modulecomprise naked eye 3D images, wherein the 3D image capture modulecomprises a 3D depth camera configured to record position information ofobjects in the images captured by the image capture module, the imagescaptured by the image capture module comprises different images for theleft eye and the right eye of the user, and the modified imagescomprises different images displayed on the display module for the lefteye and the right eye of the user, wherein the step of capturing theimages by the image capture module, the step of tracking the user'sviewpoint position, the step of adjusting the viewport within the imagescaptured by the image capture module in accordance with the user'sviewpoint position for generating the modified images, and the step ofdisplaying the modified images on the display module are performedsimultaneously, wherein the images captured by the image capture moduleare trimmed in order to be displayed on the display module and blendwith the background scene seen by the user at the same time when thedistance between the user and the portable device is reduced.